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Abstract 

Despite repeated discussions of the dangers of covariance corrections, the use of analysis 
of covariance (ANCOVA) continues to be employed in situations that should preclude its 
use. Because the use of ANCOVA is particularly appealing in much counseling research 
(where groups are often intact and sample sizes are often small for treatment 
interventions), the purpose of the present paper was to examine the use (and misuse) of 
ANCOVA in the counseling literature. Recommendations for improved ANCOVA 
behavior are presented. 
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Current Use (and Misuse) of ANCOVA in Counseling Research 
Researchers often find themselves at the mercy of intact groups (e.g., classrooms, 
schools, etc.) that limit, if not eliminate, the possibility of conducting true, randomized 
experimental research (Henson, 1998). For example, given a particular counseling 
intervention, it is often not possible to truly randomly assign clients to treatment and 
control groups. Ethical considerations also may limit the ability to deprive one group of 
clients from a potentially beneficial intervention. This also is a serious problem in 
educational circles, and accordingly, there remains a great need for experimental 
validation of educational programs in general (cf. Welch & Walberg, 1974). 

One result of this dynamic is the tendency for researchers to either (a) use existing 
groups (e.g., two fifth grade classrooms) and attempt to statistically control for 
preexisting differences between groups or (b) use one treatment group and attempt to 
statistically control for preexisting variables that may impact the final outcome. These 
types of analyses are all related, and go by various names such as analysis of covariance 
(ANCOVA) and part and partial correlation. 

In reviews of educational research literature analyses, ANCOVA appeared in 
about 4 % of the published literature (cf., Elmore & Woehlke, 1988; Goodwin & 
Goodwin, 1985). A more recent review indicated that ANCOVA appeared in 7% of the 
articles reviewed (Keselman, Huberty, Lix, Olejnik, Cribble, Donahue, Kowalchuk, 
Lowman, Petoskey, & Levin, 1998), suggesting the continued (perhaps increased) use of 
ANCOVA. In addition, Thompson (1994a) found more frequent use among doctoral 
students. The use of ANCOVA appears to have either remained relatively stable or 
slightly increased in recent years. 
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Like most analytic methods, it is important to realize that these corrections have 
certain benefits and limitations. ANCOVA, for example, is generally inappropriate when 
used with intact groups. And yet, ANCOVA does continue to be utilized in such 
situations. As Henson (1998) explained. 

Perhaps the lingering use of ANCOVA is due to the mystical promise that 
ANCOVA is a statistical correction of all pre-treatment problems and that it will 
provide increased power against Type II error. Such an argument is particularly 
compelling to doctoral students who find themselves aggressively seeking and 
even praying for statistically significant results! Unfortunately, ANCOVA has 
multiple assumptions that must be met before it can be accurately utilized, (p. 4) 
Several researchers have documented and discussed the limitations of ANCOVA 
including Henson (1998), Hines and Foil (2000), and Loftin and Madison (1991). 
Nevertheless, the use of ANCOVA persists in situations when it is inappropriate and its 
assumptions have not been met. 

Because the use of ANCOVA is particularly appealing in much counseling 
research (where groups are often intact and sample sizes are often small for treatment 
interventions), the purpose of the present paper was to examine the use (and misuse) of 
ANCOVA in the counseling literature. Three counseling journals were examined for (a) 
the frequency of use of ANCOVA, (b) whether important assumptions were addressed 
(particularly homogeneity of regression), and (c) possible misinterpretation of results 
stemming from misuse of the analyses. Examples of best and poor practice are offered 
and recommendations are presented for improved use of ANCOVA. 
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Important Considerations When Using ANCOVA 

Researchers often contuse the concepts of methodological design and statistical 
analysis. These two concepts are related but separate issues in conducting quality 
research (Henson, 1998). ANCOVA often is used as a statistical way to equate two or 
more groups when the researcher is unable to randomly assign participants to treatment 
and control groups. Thompson (1994a) stated that some researchers appear to believe that 
ANCOVA magically corrects for pre-existing group differences. Researchers often use 
ANCOVA as a way of dealing with methodological design flaws (Henson, 1998), albeit 
unavoidable flaws in many instances. 

Loftin and Madison (1991) stated that ANCOVA received favorable treatment in 
Campbell and Stanley (1963), which may have influenced researchers to regard 
ANCOVA as a panacea for many research problems. Later, Campbell and Erlebacher 
(1975) clarified statements made in the previous publication. Campbell and Erlebacher 
argued that ANCOVA is rarely useful because the more a researcher needs the controls of 
ANCOVA, the more biased the outcomes of the analysis. Unfortunately, many 
researchers cite the earlier Campbell publication and fail to mention the latter one. 

In feet, a methodological design is either experimental or not (Henson, 1998). In 
nonexperimental studies where random assignment is not used, the use of a statistical tool 
such as ANCOVA does not transform the study into an experimental one. When 
participants are not randomly assigned to groups, ANCOVA often is not being used as 
originally intended. Researchers who use existing groups in their studies know that their 
studies are at a greater risk for sampling error. If a researcher conducts a study that also 
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has a small sample size, the risk obviously increases (Huck, 2000). Unfortunately, 
ANCOVA does not dissolve the sampling present in a study. 

Importantly, ANCOVA only accurately adjusts for pretreatment group differences 
when important methodological assumptions are met (Huck, 2000; Thompson, 1994a). 
These data assumptions include the homogeneity of regression, homoscedasticity, 
homogeneity of variance (after the covariate is removed), and normality of the dependent 
variable. Violating these assumptions can affect the Type I error rate (Keselman, et al., 
1998). A more thorough summary and discussion of these assumptions can be found in 
Loftin and Madison (1991) and in Shavelson (1996). 

What ANCOVA Is 

According to Henson (1998), ANCOVA is a “regression of a covariate variable 
on the dependent variable from the entire sample ignoring group membership, at least if 
ANCOVA assumptions are perfectly met” (p. 7). Thompson (1994a) stated that 
ANCOVA residualizes the “dependent variable of all the variance that is linearly 
predictable with the covariate variable(s). Then the resulting ‘error’ or ‘e’ scores are used 
as the new dependent variable in an ANOVA” (p. 25). The intent of ANCOVA is to 
“assign a portion of the variance in the dependent variable that would normally be 
attributed to error in a regular analysis of variance (ANOVA) to an extraneous co variate 
variable” (Henson, 1998, p. 7). 

When applied appropriately, ANCOVA can provide increased power against 
Type II error. However, the methodological assumptions that are required to be met when 
using ANCOVA are rarely satisfied (Henson, 1998; Loftin & Madison, 1991). 
ANCOVA’s promise of increased power against Type II error is only fulfilled in cases in 




7 



ANCOVA 7 



which the covariate is uncorrelated with the independent variable. In fact, the covariate 
may rob the independent variable of some the variance attributed to it if the two variables 
are correlated (Henson, 1998). As Thompson (1994a) stated: 

When the covariate is related to the treatment variable, use of the covariance 
correction will alter the effects attributed to the treatment itself. For example, one 
might have a very effective intervention that looks completely ineffectual, 
because the covariate is given credit for the variance that would correctly 
otherwise be attributed to the treatment variable, (p. 27) 

Thus, the use of the ANCOVA correction may actually decrease power against Type II 
error if the covariate robs the treatment variable of its effect. Of course, researchers can 
verify the magnitude of the relationship between the covariate and the grouping variables 
by finding the correlation between them. If these variables are correlated, the effects 
attributed to the treatment effect could be altered. 

Although it is seldom examined, an easy to evaluate the relationships between the 
covariate and dependent variable and the covariate and the grouping variable would be to 
simply calculate the Pearson r between the variables or examine some index of effect size 
(e.g., perhaps use r\ 2 for the effect between a continuous covariate and categorical 
grouping variable). This would help researchers to evaluate whether the ANCOVA might 
actually reduce power against Type II error. 

Some Important (Often Overlooked! Assumptions Related to ANCOVA 

As previously stated, several methodological assumptions must be met in order 
for ANCOVA to be used accurately. As demonstrated by Henson (1998), ANCOVA is 
essentially a two-stage analysis. In the first stage, the dependent variable is regresses on 
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the covariate (s) ignoring group membership as one would in a traditional regression 
analysis. Therefore, all of the assumptions related to regression, such as 
homoscedasticity, would be applicable for this stage of the analysis. 

In the second stage, a traditional ANOVA is performed on the residualized 
dependent variable (i.e., the error scores that were not predictable by the co variate). 
Therefore, all the assumptions of a traditional ANOVA would apply here, including 
homogeneity of variance between groups and normalcy of the dependent variable 
(particularly when sample sizes are small) (Huck, 2000). 

These expectations notwithstanding, ANCOVA has an additional assumption that 
serves as a bridge of sorts between the two stages. This assumption is called homogeneity 
of regression and will be discussed briefly here. Thompson (1994a, p. 28) noted that 
statisticians use the name homogeneity of regression assumption because the phrase 
sounds fancier than the “equality of the B weights” assumption. If the groups used in 
ANCOVA were not created through random assignment (which is often the case), the B 
weight relationships between the covariate and the dependent variable often are not 
equivalent across the groups. 

The goal of ANCOVA is to equate groups on some variable before analyzing the 
treatment effects. When the regression of the covariate on the dependent variable is 
performed, group membership is ignored . If one regression equation is used for all groups 
(as it is in ANCOVA), the equation must be representative of each of the groups 
examined individually, otherwise the use of the single equation will distort the 
residualized dependent variable scores (Henson, 1998). Intuitively, we know that if the 
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regression slopes are notably different in the individual groups, the pooled regression 
slope does not accurately represent any of the groups. 

If researchers do not examine the slopes of the individual groups in order to 
determine if they are (roughly) the same, these researchers risk applying ANCOVA to 
unequivalent groups and not representing any group well (Loftin & Madison, 1991). 
Loftin and Madison stated that the homogeneity of regression assumption “requires that 
the relationship between the dependent variable Y and the covariate X is constant within 
all k levels of the independent variable (K)” (p. 141). Because of its importance, Huck 
(2000) noted, the homogeneity of regression assumption “should always be tested when 
ANCOVA is used” (p. 552, emphasis in original). 

Method 

Article Selection 

In the current study, we examined the use of ANCOVA in three counseling 
journals from 1 997-2001 . All articles appearing in the Journal of Counseling and 
Development ( JCD ), Counselor Education and Supervision (CES), and the International 
Journal of Play Therapy ( IJPT) were selected for this study. JCD is the official journal of 
the American Counseling Association with a circulation of over 50,000. CES and IJPT 
are specialty journals with greater focus on counselor education and practice. Book 
reviews, practice-based articles, and editorial addresses were deleted. 

Articles were coded into categories as reflected in Table 1. In previous studies, 
ANCOVA was used in 4-7% of the articles reviewed (cf. Elmore & Woehlke, 1988; 
Goodwin & Goodwin, 1985; Keselman et al., 1998). In the current study, we found that 
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ANCOVA was used in 21 studies, which was 8% of the 250 articles reviewed. As Table 
1 indicates, ANCOVA was more popular in the JCD and IJPT than it was in CES. 



INSERT TABLE 1 ABOUT HERE 

The variables that we utilized in analyzing the ANCOVA’s are presented in Table 
2. We included every application utilized in the ANCOVA articles; therefore, the number 
of applications of ANCOVA reflects higher frequencies than the total number of 
ANCOVA articles. Several researchers conducted multivariate and univariate 
applications in the same study. We examined each application of ANCOVA to see how 
researchers addressed (a) design characteristics, (b) assignment of participants to groups, 
(c) the selection of covariate(s), (d) assumptions, and (e) use of effect sizes to describe 
relationships between the covariate and the dependent variable, covariate and the 
grouping variable, and main/interaction effects. 



INSERT JABLE 2 ABOUT HERE 

Design 

As Table 2 indicates, researchers used one-way designs in the vast majority (87%) 
of their applications of ANCOVA and MANCOVA. Univariate uses (94%) were much 
more popular than multivariate (6%). In all five uses of MANCOVA, the multivariate 
analysis was followed post hoc with a univariate test. Although it is beyond the scope of 
this paper to discuss the issue fully, it is inconsistent to first declare interest in a 
multivariate system of variables then use univariate tests to evaluate the contributions of 
the dependent variables (cf. Fish, 1988; Henson, 1999; Thompson, 1994b). A better 
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procedure would be to follow factorial MANCOVA with another multivariate procedure 
such as discriminant analysis (Huberty, 1994). 

Researchers also used unbalanced designs (86%) much more than they used 
balanced designs. This finding was higher than the percentage of unbalanced designs 
(75%) found by Keselman et al. (1998). In the current study, however, some of the 
studies used cells that were close to balanced. Ray and Altekruse (2000), for example, 
used a one-way design with three levels. The cell sizes were 22, 20, and 22. On the other 
hand, the cells of iEgisdottir and Gerstein (2000) in their three-way design were very 
unbalanced. Their cells were nationality (261, 225), gender (150, 334), and personality 
type (299, 66, 30). As Keselman et al. (1998) have stated, when group sizes are unequal, 
the effects of validity assumption violations can be exacerbated. In some studies, basic 
descriptive statistics and cell sizes were not even given. In Crutchfield and Borders 
(1997), the researchers included information about the three groups of supervisees, but 
did not include information about cell sizes for client groups and gender groups. 

As stated previously, many researchers have cited Campbell and Stanley (1963) 
as a rationale for utilizing a “quasiexperimental”, nonequivalent group design. For 
example, Harris and Landreth (1997) stated that, “a pretest-posttest, nonequivalent 
control group design (Campbell & Stanley, 1963) was used in this study ...” (p. 61). As 
Loftin and Madison (1991) mentioned, however, Campbell and Erlebacher (1975) 
clarified the limitations of ANCOVA that were not specifically addressed in the earlier 
publication. 

Unfortunately, most of the ANCOVA applications that we examined were used 
with intact groups (86%). By contrast, Keselman et al. (1998) found that 67% of the 
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ANCOVA studies they reviewed used nonrandomized experimental units. As Henson 
(1998) and Loftin and Madison (1991) have cautioned, using ANCOVA with intact 
groups is problematic. ANCOVA does not correct for all of the pre-treatment differences 
that researchers often believe it does. In some of the studies that we examined group 
assignment was ambiguous. In Chau and Landreth (1997), the researchers stated that 
participants “were assigned to a control group or an experimental group according to 
their work or school schedule” (p. 78). In the next paragraph, Chau and Landreth reported 
that participants were “randomly selected into the experimental group” (p. 79). 
Assumptions 

As Table 2 indicates, researchers only verified the homogeneity of regression 
assumption in 17 of the 86 applications (20%). This percentage is similar to Keselman et 
al. (1998), where 18% of the studies utilizing ANCOVA commented on the homogeneity 
of regression assumption. All of the researchers in the current study who verified the 
homogeneity of regression assumption utilized statistical significance testing to 
determine whether the slopes were parallel (equal). For example, Sullivan and Mahalik 
(2000) stated that, “tests for regression parallelism were calculated. Results for these tests 
indicated that there were no significant age by condition interaction effects on the pre- 
and posttest CDMSE and VEC scores” (p.59). One limitation of using this approach to 
check the equality of the slopes is that statistical significance testing is heavily impacted 
by sample size (Cohen, 1994; Daniel, 1998; Henson & Smith, 2000; Thompson, 1996; 
Thompson & Snyder, 1998). 

For example, gross slope differences could be overlooked with small sample sizes 
because the differences may not be statistically significant. With large sample sizes. 
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trivial differences in slope could yield statistically significant results even though the 
differences in slope are not practically different. Because many of the articles examined 
in the current study had very small sample sizes, it is possible that even if a researcher 
verified the homogeneity of regression assumption with a statistical significance test, the 
researcher could still be dealing with notable differences in the slopes. This unintentional 
omission could distort the results of the ANCOVA. 

Loftin and Madison (1991) recommended examining the homogeneity of 
regression by using a scatterplot of the association instead of a test of statistical 
significance. A scatterplot allows the researcher to visually inspect whether the slopes are 
reasonably parallel across all of the groups and is consistent with calls for increased 
graphical examination of our data (Wilkinson & APA Task Force on Statistical Inference, 
1999). 

As Table 2 indicates, none of the other validity assumptions that we examined 
were verified. Specifically, researchers in this study did not verify the assumptions of 
normality and homogeneity of variance, which also were given little attention in the 
Keselman et al. (1998) review of ANCOVA. These authors found that only 4 % of the 
ANCOVA articles that they examined considered the assumption of normality and only 
9 % of the studies examined commented on homogeneity of variance. 

Relationships Between Study Variables: Effect Sizes 

Authors reported any kind of effect size in the current study in only 3 of 86 (4%) 
of the applications of ANCOVA. The only effect sizes that were reported were for main 
treatment effects or interaction effects, and all were standardized mean differences. In 
only one case did the researcher also interpret the magnitude of the effect size. 
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The lack of effect reporting in these articles is not consistent with current 
methodological trends (Daniel, 1998; Thompson, 1996; Thompson & Snyder, 1998). 
Indeed, the fifth edition of the American Psychological Association Publication Manual 
(2001) called the “failure to report effect sizes” a “defect in the design and reporting of 
research” (p. 5). The Manual later observed: “For the reader to fully understand the 
importance of your findings, it is almost always necessary to include some index of effect 
size or strength of relationship in your Results section” (p. 25). Fortunately, some 
journals such as JCD now require effect reporting. Readers are referred to a recent JCD 
article (Thompson, in press) for more information regarding various kinds of effects and 
result significance. 

In some cases, the reporting of effect sizes would have provided greater clarity on 
the “practical” significance of authors’ findings. In Crutchfield and Borders (1997), for 
example, the researchers stated that, “these individually nonsignificant results showed 
movement in the preferred direction each time, indicating small but pervasive effects of 
treatment” (p. 226). Reporting of effect sizes would have allowed evaluation of the 
magnitude of these effects. Similarly, Tyndall-Lind, Landreth, and Giordano (2001) 
stated, “the results from this study strongly point to the effectiveness of intensive sibling 
group therapy in a variety of areas with child witnesses of domestic violence” (p. 67). 
Effect sizes could have quantified how “strongly” the treatment impacted the children in 
the study. 

Important for ANCOVA use are the relationships between the covariate and the 
dependent and grouping variables. As Table 2 indicates, none of the researchers reported 
the correlations between the covariates and the grouping variables. This finding was 
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unfortunately expected, of course, and comparable to Keselman et al. (1998), who stated 
that the researchers that they reviewed “generally ignored the interaction effects between 
the covariate(s) and the grouping variables” (p. 376). This effect size relationship is 
important because if the grouping variables are correlated with the covariate, the initial 
regression of the covariate can remove parts of the sum of squares explained by the 
grouping variables (Loftin & Madison, 1991). Therefore, the researchers in the current 
study were seemingly unaware of how much their covariates may have robbed the effect 
attributed to their treatment variables. 

Further, the relationship between the covariate and the dependent variable was 
never reported. Because the covariate removes variance from the dependent variable, the 
magnitude of this relationship is critical to understanding the remaining residualized 
variable. In cases when the covariate has a strong relationship with the dependent 
variable, Henson (1998) suggested that, “one must question what is left of 
the... dependent variable after the dramatic impact ofthe covariate” (p. 15). As 
Thompson (1992) explained, “Statistical corrections remove parts of the dependent 
variable and then analyze whatever’ s left, even if whatever’ s left no longer makes any 
sense. At some point, we may no longer know what it is we’re analyzing” (pp. xiii-xiv). 
Given that (a) 85% of the covariates used were simply pre-test versions ofthe dependent 
variable and (b) that we might hope for reasonable test-retest reliability between pre- and 
post-testing (e.g., moderate to strong relationships between the covariate and dependent 
variable scores), it is not unreasonable to suspect that in many cases the studies examined 
here analyzed a residualized dependent variable that may not have possessed the same 
degree of validity as the original scores. 
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Summary and Recommendations 

Our examination of the 21 articles that reported 86 applications of ANCOVA 
revealed that these counseling researchers primarily used univariate applications of 
ANCOVA. The researchers generally used intact groups and unbalanced cells in most of 
their studies. In addition, most did not comment on some basic assumptions associated 
with ANCOVA including the very important homogeneity of regression assumption. 
Keselman et al.’s (1998, p. 376) claim that, “researchers appear to be unaware of, or at 
least fail to recognize, the assumptions that underlie the statistical models that they use” 
appears applicable to the current selection of ANCOVA articles. 

We also found that effect sizes were ignored by most of the researchers reviewed. 
The relationship between the covariate (s) and the grouping variable (s) was completely 
overlooked, as was the relationship between the covariate (s) and the dependent 
variable (s). 

Based on our review of the use (and misuse) of ANCOVA in the counseling 
literature examined here, we recommend the following: 

1 . Verify the homogeneity of regression assumption. As Huck (2000) stated, the 
homogeneity of regression assumption needs to be verified in every use of 
ANCOVA. ANCOVA cannot be used appropriately unless regression slopes are 
parallel (Loftin & Madison, 1991). 

2. Utilize scatterplots to verify the homogeneity of regression assumption. If 
researchers use statistical significance tests to examine this assumption, they can 
misinterpret the results if the sample size is small or large (Loftin & Madison, 
1991). Scatterplots allow the researchers to visually inspect the regression slopes. 
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3. Report treatment and interaction effect sizes. In this study, effect sizes were rarely 
reported. Effects allow researchers to evaluate more clearly (and communicate to 
the reader) the practical significance of their results. 

4. Examine and report the correlation between the covariate (s) and the grouping 
variable (s). If the covariate and the grouping variables are correlated, the 
covariance correction may alter the effects attributed to the treatment. 

5. Examine and report the correlation between the covariate (s) and the dependent 
variable (s). The magnitude of this relationship helps with the interpretation of the 
residualized dependent variable. 

6. Utilize reasonably balanced cell designs with ANCOVA. Utilizing excessively 
disproportionate group sizes may raise concern about the statistical validity of 
one’s findings (Keselman et al., 1998). 

Henson (1 998) noted that, “As with most statistical analyses, ANCOVA can be 
reasonably applied when its assumptions have been met” (p. 1 5). However, it is 
problematic that, generally, ANCOVA is most desired as a statistical correction when the 
study features most preclude its use, such as when using unequal, intact groups. At a 
minimum, authors are encouraged to apply ANCOVA with caution. These 
recommendations may be useful in helping counseling researchers thoughtfully employ 
this method. 
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Table 1 

Types of Articles Reviewed Including ANCOVA 



Journal 



Type of article 


JCD 


CES 


IJPT 


Total 


Descriptive 


1 


13 


4 


24 


Inferential (ANCOVA) 


68(9) 


27(2) 


19(10) 


114 


Qualitative /theory 


63 


32 


17 


112 


Total articles reviewed 3 


138 


72 


40 


250 



a Articles that were practice-based were excluded. 
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Table 2 

Summary of ANCOVA Applications (n=86) b 



Design 


n 


Grouping 


n 


Covariates 


n 


One Factor 


75 


Intact 


74 


Nature : 




Two Factor 


1 


Matching 


4 


Pre-test 


73 


Three factor 


10 


Random Assignment 


8 


External 


13 


Univariate 


81 










Multivariate 


5 










MANCOVA/ANCOVA post hoc 


4 






Number : 




MANCOVA/ANOVA post hoc 


1 






Single 


85 


Balanced 


12 






Multiple 


1 


Unbalanced 


74 











Assumptions/Effect Sizes 


Assumptions 


n 


Effect Sizes/Relationships 


n 


No comment 


69 


No comment on effect 


83 


Homogeneity of reg. 


17 


Correlation b/t Covariate/DV 


0 


Homoscedasticity Cov/DV 


0 


Correlation b/t Cov/Group Var. 


0 


Homogeneity of variance 


0 


Main effect /interaction effect 


3 


Normality 


0 







b Totals are greater than the number of articles because the total 
number of ANCOVA applications were examined. 
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